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The  JNTF  has  defined  a  cost-effective,  tailorable,  and  scalable  V&V 
methodology  which  this  past  year  was  effectively  used  on  the  first  release  of  a 
missile  defense  system  simulation.  We  will  present  the  lessons  learned  from 
applying  the  methodology  on  the  National  Missile  Defense  (NMD)  High  Fidelity 
System  Simulation  (HFSS).  This  effort  was  presented  with  numerous  technical 
and  programmatic  challenges,  and  the  adaptability  of  the  JNTF  V&V 
Methodology  contributed  to  the  success  of  the  effort. 


The  HFSS  is  a  missile  defense  simulation  that  covers  all  aspects  of  performance 
modeling  from  threat  launch,  space-based  tracking,  ground  radar  tracking,  battle 
planning,  and  interceptor  launch.  It  is  intended  to  support  system  engineering 
studies  and  analyses  while  the  system  is  in  the  early  requirements  definition  and 
system  design  phases. 


The  methodology  produced  a  V&V  report  that  provided  a  well  received 
assessment  of  the  capabilities  and  limitations  of  the  simulation.  It  is  clear  that 
the  use  of  a  well  defined  and  adaptable  process  contributes  to  the  quality  of  the 
product  while  addressing  the  numerous  technical  and  programmatic  challenges. 
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Overview 

♦  The  National  Missile  Defense  (NMD) 

High  Fidelity  System  Simulation  (HFSS) 

♦  JNTF  V&V  Methodology  Overview 

♦  Tailoring 

♦  Lessons  Learned 

♦  Summary 
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Models,  simulations,  and  testbeds  are  now  an  integral  part  of  weapon  systems 
design  and  development  as  well  as  test  and  evaluation.  The  ultimate 
effectiveness  of  weapon  systems  therefore  depends  more  and  more  on  the 
validity  of  models  used  in  their  acquisition.  Over  the  past  ten  years,  the  Joint 
National  Test  Facility  (JNTF)  has  developed  a  methodology  for  the  Verification 
and  Validation  (V&V)  of  models,  simulations  and  testbeds  and  has  evolved  it 
using  insights  gained  through  its  use  in  numerous  assessments.  This  presentation 
describes  how  the  JNTF  V&V  methodology  was  used  on  the  National  Missile 
Defense  (NMD)  High  Fidelity  System  Simulation  (HFSS). 

A  description  of  HFSS  will  be  provided  to  establish  the  context  for  this  project. 

A  overview  of  the  JNTF  V&V  methodology  will  provide  greater  understanding 
of  the  breadth  of  procedures  available  within  the  methodology.  A  discussion  of 
the  tailoring  will  show  how  specific  procedures  can  be  selected  based  on  cost  or 
schedule  constraints.  The  lessons  learned  provide  insights  that  are  applicable  to 
many  V&V  efforts,  and  may  provide  help  to  other  V&V  practitioners  in  the 
missile  defense  community. 
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HFSS  Overview 
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♦  HFSS  is  an  end-to-end  NMD  simulation  (Threat 
Launch  to  Negation  or  Impact) 


♦  HFSS  intended  use 

-  Systems  engineering  studies 

-  System  performance  verification 

♦  Developed  by  the  NMD  Systems  Engineering 
Contractor 

♦  Verification  and  Validation  performed  by  the  JNTF 
and  the  Naval  Surface  Warfare  Center 
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HFSS  provides  an  end-to-end  simulation  of  the  National  Missile  Defense 
architecture.  It  was  intended  to  be  used  by  the  Systems  Engineering 
organization  to  study  architecture  issues,  requirements  trades,  and  to  evaluate  the 
performance  of  the  proposed  system. 

The  simulation  was  built  by  the  NMD  Systems  Engineering  and  Integration 
Contractor.  The  V&V  was  assigned  to  the  JNTF  with  support  from  the  Naval 
Surface  Warfare  Center,  Dahlgren  Division. 
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This  diagram  shows  the  scope  of  the  NMD  system  as  simulated  in  the  HFSS. 
HFSS  simulates  the  launch  of  threats  using  pre-approved  threat  data.  These 
threats  are  then  detected  by  a  Space  Based  Infra-Red  System  (SBIRS),  which 
provides  initial  warning  messages  to  the  Battle  Manager.  Later  detection  and 
tracking  is  performed  by  the  early  warning  radars,  which  then  provide  queueing 
to  the  tracking  radar.  Based  on  the  satellite  and  radar  data,  the  battle  manager 
develops  an  engagement  plan,  and  then  tasks  the  weapons  to  launch  exo- 
atmosheric  kill  vehicles  to  destroy  the  threat. 

The  HFSS  release  used  in  this  effort  implemented  only  low-fidelity  versions  of 
the  element  models.  As  a  first  release,  it  was  built  to  establish  the  overall 
simulation  architecture.  Future  plans  called  for  incorporation  of  high-fidelity 
models,  but  a  change  in  contractors  resulted  in  no  further  work  on  the  HFSS. 
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Tailoring  provides  the  greatest  level  of  confidence  to  the  customer,  within  the 
constraints  of  the  project.  Careful  review,  and  interaction  with  the  customer, 
results  in  a  V&V  effort  which  provides  an  acceptable  level  of  confidence  while 
accounting  for  the  resource  constraints. 

The  contraints  placed  on  this  particular  effort  were  severe,  and  only  allowed  for 
a  limited  evaluation  of  the  simulation. 

The  most  critical  constraint  effecting  this  effort  was  the  need  for  an  early 
assessment  of  the  HFSS  capabilities.  The  customer  wanted  to  know  at  an  early 
date  the  state  of  the  simulation  and  its  applicability  to  potential  users.  This 
meant  that  the  V&V  effort  was  based  on  a  preliminary  release,  built  to  the 
evolving  requirements  of  the  proposed  NMD  architecture.  This  also  meant  that 
the  documentation  was  incomplete,  so  the  primary  artifact  available  for 
evaluation  of  the  system  was  the  software  itself  (that  is,  there  were  no  design 
materials). 

Another  constraints  was  the  use  of  two  government  organizations  for  the  V&V. 
The  meant  that  tasking  had  to  be  worked  out  between  the  JNTF  in  Colorado  and 
NSWC  in  Virginia,  and  that  the  team  had  to  remain  coordinated  and  in  regular 
communication. 
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(HH  JNTF  V&V  Methodology 

'HP'  39  Procedures  Available 

Conceptual 

Validation 

Software 

Verification 

Operational 

Validation 

Data 

Validation 

•  Development  history 
analysis 

•  Model  derivative  analysis 

•  Previous  model  use 
analysis 

•  Requirements  criticality 
analysis 

•  System  analysis 

•  Modeling  concepts  analysis 

•  Input/output  fidelity  analysis 

•  Algorithm  analysis 

•  Logic  trace  analysis 
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•  CASE  and  design 
methodology  adherence 
analysis 

•  Screening  procedure 

•  Process  metrics  analysis 

•  Products  metric  analysis 

•  Internal  software  testing 
analysis 

•  Code  analysis 

•  Security  code  analysis 

•  QA  review 

•  Documentation  review 

•  User  support  review 

•  Model  flexibility  assessment 

•  CM  Review 

•  IV&V  Review 

•  Graphical  display  evaluation 

•  Animation  test 

•  Fixed  value  test 

•  Simplified  assumption 
testing 

•  Extreme  condition  testing 

•  Predictive  validation  testing 

•  Output  validation  analysis 

•  Comparison  to  test  data 

•  Sensitivity  analysis 

•  Feedback  loop  analysis 

•  Event  sequencing  testing 

•  Head  to  head  comparison 

•  Input/output  relationship 
analysis 

•  Input  data  analysis 

•  Data  consistency 
analysis 

•  Portrayal  of 
constants  analysis 

•  Distribution  form 
analysis 

The  JNTF  has  been  performing  V&V  on  models  and  simulations  used  for 
Ballistic  Missile  Defense  (BMD)  since  1988.  The  procedures  used  have  been 
documented  and  institutionalized  so  that  future  projects  could  spend  more  time 
on  engineering  and  less  on  developing  procedures  and  so  that  the  JNTF  V&V 
efforts  would  be  repeatable.  Over  time  39  procedures  have  evolved  and  proven 
to  be  scalable  to  a  variety  of  V&V  efforts.  Although  there  is  not  a  one-to-one 
correspondence  between  the  JNTF  procedures  and  those  in  the  newly  released 
DMSO  Recommended  Practices  Guide,  there  is  a  close  correspondence.  In 
general,  the  JNTF  procedures  have  been  documented  in  substantially  more  detail 
than  those  of  DMSO. 
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iMgk  JNTF  V&V  Methodology 

^lll|r  4  Procedures  Selected 

Conceptual 

Validation 

Software 

Verification 

Operational 

Validation 

Data 

Validation 

•  Development  history 
analysis 

•  Model  derivative  analysis 

•  Previous  model  use 
analysis 

•  Requirements  criticality 
analysis 

•  System  analysis 

•  Modelinq 
concepts 
analysis 

•  Input/output  fidelity  analysis 

•  Algorithm  analysis 

•  Logic  trace  analysis 

UNCLASSIFIED 

•  CASE  and  design 
methodology  adherence 
analysis 

•  Screening  procedure 

•  Process  metrics  analysis 

•  Products  metric  analysis 

•  1  nternal  software  testing 
analysis 

•  Code  analysis 

•  Security  code  analysis 

•  QA  review 

•  Documentation  review 

•  User  support  review 

•  Model  flexibility  assessment 

•  CM  Review 

•  IV&V  Review 

•  Graphical  display  evaluation 

•  Animation  test 

•  Fixed  value  test 

•  Simplified  assumption  testing 

•  Extreme  condition 
testina 

•  Predictive  validation  testing 

•  Output  validation  analysis 

•  Comparison  to  test  data 

•  Sensitivity  analysis 

•  Input  data 
analysis 

•  Data  consistency 
analysis 

•  Portrayal  of 
constants  analysis 

•  Distribution  form 
analysis 

•  Feedback  loop  analysis 

•  Event  sequencing  testing 

•  Head  to  head  comparison 

•  Input/output  relationship 
analysis 

Of  the  39  procedures,  4  were  selected  for  use  on  the  HFSS. 

Modeling  Concepts  Analysis  is  intended  to  be  a  review  of  the  model 
implementation  to  assess  whether  or  not  the  selected  methodology  elements  are 
consistent  with  the  goals  for  model  application  as  represented  by  the  intended 
use  statement.  In  particular,  fidelity,  repeatability,  data  availability,  and  code 
capability  are  reviewed. 

Knowledge  of  model  behavior  under  extreme  input  conditions  is  important  not 
only  at  the  time  of  model  selection  and  data  entry,  but  also  as  an  indication  of 
how  adaptable  the  model  may  be  to  other  non-design  uses.  This  procedure  will 
investigate  the  nature  of  input,  model  behavior  under  extreme  conditions,  and 
how  thoroughly  and  adequately  this  behavior  was  documented. 

Sensitivity  Analysis  examines  a  model's  sensitivity  to  incremental  changes  in 
input  or  internal  variables.  Any  variable  to  which  the  model  is  particularly 
sensitive  is  examined  more  closely  to  determine  if  it  has  been  modeled  correctly. 

Input  Data  Quality  Review  determines  if  the:  (1)  input  data  supports  the  intended 
application,  (2)  source(s)  of  input  data  is/are  credible  or  officially  sanctioned, 
and  (3)  input  data  is  reliable  and  current.  Input  Data  Quality  Review  focuses  on 
the  model  at  the  vertical  level,  examining  a  particular  aspect  of  the  model  in 
detail. 
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JNTF  V&V  Procedures 


Specific  Procedures: 


Model  Concepts  Analysis 

Input  Data  Analysis 

Threat  SBIRS  Interceptor 

Threat  Scenario  1 

UEWR  XBR  BMC3 

IFICS  Environment 

Extreme  Condition 

Sensitivity  Analysis 

Ideal  Performance 

Interceptor  SBIRS 

Degraded  Performance 

Battle  Manager  Radar 
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Each  of  the  four  procedures  was  then  executed  for  specific  models  or  conditions. 

The  Models  Concepts  Analysis  was  performed  for  each  major  sub-system  modeled  in  the 
HFSS.  This  was  a  detailed  review  of  the  design  and  implementation  of  the  model. 

The  HFSS  organizes  it  databases  around  threat  scenarios,  although  it  also  includes  all  of  the 
data  needed  for  all  of  the  simulated  sub-systems.  In  this  case,  Threat  Scenario  1  was  built  by 
the  developer,  and  then  used  to  develop  all  other  test  case  databases.  Verification  of  this 
database  provided  confidence  in  all  of  the  other  test  case  databases. 

Because  of  constraints  on  time,  system  level  extremes  were  performed.  In  the  ideal  case,  all 
weapons  and  sensors  worked  in  an  ideal  manner.  Probabilities  were  set  to  1.0  to  enable  ideal 
results.  The  degraded  run  provided  failures  on  individual  sub-systems  to  demonstrate  the 
impacts  at  the  system  level. 

Sensitivity  analyses  were  performed  for  the  to  evaluate  model  performance  in  4  key  areas: 
interceptor,  SBIRS,  Battle  Manager,  and  Radar. 
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Lessons  Learned 


Communications.  VTCs  and  Telecons.  E-Mail  Relay. 
Regular  Visits  and  team  coordination  meetings.  On¬ 
site  presence  at  the  development  site 


Computer  Resources.  Start  early  on  acquisition  and 
configuration  of  H/W  and  S/W  (especially  COTS). 

V&V  Readiness  Review. 

Pause  to  assess  what  you  have  accomplished, 
and  review  what  lies  ahead. 

Pre-defined  Methodology  lets  you 
focus  on  the  engineering  problems. 
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Communications  is  critical  to  a  team  effort,  and  it  is  even  more  critical  when 
geographically  dispersed.  Regular  team  meetings  were  held  each  week  via 
telecon  or  video  teleconferencing.  We  didn’t  always  have  a  planned  agenda,  but 
it  was  still  valuable  to  get  together.  Also,  regular  site  visits,  at  least  monthly, 
provided  an  opportunity  for  more  detailed  communication  and  coordination. 

Computer  resources  for  a  Y&V  team  are  essential.  We  started  early  in  our 
acquisition  process.  Just-in-time  is  most  likely  a-little-too-late.  It  was  important 
to  try  out  hardware  and  software  configurations  before  critical  need  dates.  The 
use  of  COTS  in  simulation  development  is  a  new  trend,  and  it  means  the  V&V 
team  must  have  a  proper  configuration.  We  found  that  supposedly  backward 
compatible  versions  are  not  always  compatible. 

We  held  a  V&V  Readiness  Review  as  we  started  our  final  V&V  effort.  At  this 
point,  we  had  been  running  the  simulation  for  2  months,  and  performing  dry-runs 
of  specific  procedures.  The  VRR  provided  a  chance  to  asses  the  breadth  and 
depth  of  our  effort  before  getting  too  close  to  our  end  date. 

The  JNTF  methodology  made  our  planning  and  execution  easier.  We  were 
focused  on  the  engineering  problems  instead  of  process  design. 
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Lessons  Learned 


Document  Production.  With  a  team,  you  have 
different  format  and  submittal  requirements. 

Give  organizational  credit. 


Final  Assessment  and  Interpretation  of 
results.  Half  empty  or  half  full  ?  The 
Intended  Use  is  critical. 


Do  not  apply  standards  for  Operational 
Software  to  a  Computer  Simulation 


im 
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Document  production  is  critical  to  having  the  body  of  evidence  needed  for 
accreditation  of  the  simulation.  It  is  important,  especially  in  a  team 
environment,  to  work  out  early  the  logistics  and  mechanics  of  document 
production  and  submission.  It  is  especially  important  to  understand  management 
review  and  approval  requirements. 

Understanding  the  Intended  Use,  especially  of  preliminary  releases,  is  important. 
We  had  ambiguities  that  led  to  disagreement  on  our  findings.  This  also  relates  to 
the  different  organizational  cultures. 

The  V&V  team  must  remember  that  a  simulation  may  not  need  to  meet  the  same 
quality  requirements  as  an  avionics  package.  It  is  a  waste  of  resources  to 
evaluate  against  requirements  that  are  too  stringent. 
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The  Art  of  V&V 


The  Art  of  War:  Its  Application  to  V&V  of  Models  and  Simulations 

♦  OBJECTIVE:  You  and  the  customer  must  agree  on  the  intended  use 

♦  PLANNING:  Prepare  clear,  uncomplicated  plans  and  procedures 

♦  OFFENSIVE:  Demand  early,  unofficial  deliveries 

♦  MASS:  People  and  computers  must  be  in  place  and  ready 

♦  ECONOMY  OF  FORCE:  Do  not  get  distracted  by  problems  or 
issues  unrelated  to  the  Intended  Use 

♦  MANEUVER:  Be  flexible  and  adaptable  to  the  technical  and 
programmatic  problems  encountered  late  in  the  process 
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While  working  this  project,  a  small  poster  summarizing  Sun  Tzu’s  “The  Art  of 
War”  was  posted  in  our  work  area.  Some  of  the  lessons  from  the  Art  of  War  are 
applicable  to  V&V.  In  this  case,  the  enemy  is  not  the  developer.  The  enemy  is  a 
bad  or  incorrect  simulation.  Applying  these  concepts  can  help  in  performing 


V&V. 
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SUMMARY 
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1 - 

♦  Procedures  can  be  selected  based  on  customer 
constraints  and  needs 


♦  A  pre-defined  methodology  means  resources 
used  for  engineering  instead  of  process  design 

♦  The  JNTF  methodology  results  in  a  high  quality 
product  even  in  a  resource  constrained 
environment 
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Having  a  set  of  procedures  defined,  documented,  and  institutionalized  provides 
clear  benefits  to  a  V&V  team.  Time  is  not  wasted  in  defining  “how”  to  proceed 
with  V&V;  the  analysts  can  quickly  focus  on  the  engineering  issues.  Specific 
techniques  can  be  selected  for  the  effort  to  minimize  risks,  or  to  increase  the 
analysis  of  critical  areas.  The  JNTF  V&V  Methodology  has  proven  to  be  an 
effective  tool  for  obtaining  a  well  balanced,  comprehensive  product  in  a  resource 
constrained  environment. 
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